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Heuristic Evaluation 

Method 

■ A Heuristic Evaluation is an expert usability inspection method for software 
applications that helps to identify usability problems with the Ul 

■ The software Ul designs were assessed against recognized usability principles 
(heuristics), mobile experience, and MOU requirements 

Nielsen’s 10 Usability Heuristics: Nielsen, Jakob. (1994). Heuristic evaluation. In Nielsen, J., and 
Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, NY 


Objectives 

■ Identify potential usability issues with the Free File software providers 

■ Identify areas where there is misalignment with the IRS requirements outlined 
in the MOU 

■ Identify issues on mobile device access and usability of the software 

■ Prioritize identified usability issues via severity ratings 

■ MOU issues were rated in detailed spreadsheets which can be viewed in Appendix H. 
The severity scores were determined based on whether the user could complete the 
task at hand. 


Severity Level & Description 

5 

No observed usability problems related to this heuristic 

4 

Nuisance problems identified 

3 

Minor usability problems 

2 

Major usability problems 

1 

Severe usability problems (possible user failure) 


Procedure 

1. Review and rate task flow and user interface of each Free File member software 
site. Every problem identified is tagged with one often Nielsen’s heuristics, 
mobile friendliness, and MOU requirements with a 1-5 severity score. 

2. When multiple issues are found under one category, the lowest score 
determined is used 

3. Individual scores are tallied and a percentage score determined 
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Heuristic Evaluation Dashboard Summary 


Dimension 


User Experience 


UX MOU 
Adherence 











10 11 12 




4 t 4 t 


4 * 4 t 4 t 


Mobile Experience 


♦ t 4 


4 t * 


* t t * 


SUMMARY 

At a glance, the Free File providers adhere 
to many UX best practices. Overall, they 
could improve the most in “Error 
Prevention” and “Match Between System & 
Real World” heuristics. The most significant 
UX MOU improvement to make across the 
board is “Transparency around ineligibility.” 


Score range 


85% to 100%. Area of strength. 

70% to 84%. Room for improvement. 
^ 0% to 69%. Area of weakness. 


All of the scores from the heuristic review are rolled up into this dashboard which shows a 
quick view of the summary scores for user experience (UX), mobile experience, and UX MOU 
adherence. The pages that follow provide a more detailed breakout of the scores from the 
heuristic review by each heuristic and grouping of MOU requirements. 



The top two highest-rated providers were 2 and 10 which all had 
exceptional user experience, intuitive mobile experiences, and adhered 
to the MOU. Other providers might consider mirroringthe transparency 
provider 10 has around free file eligibility, which lets the user know very 
quickly whether or not they are eligible to proceed through the Free File 
process. The colloquial language and step-by-step flow of provider 2 
was a refreshing user experience, especially for novice taxpayers. 



The providers that received the lowest ratings were 5 and 9 due to 
difficult to use interfaces. Providers 7,9, and 12 did not have mobile- 
optimized sites, making it very difficult for users to file their taxes from 
a mobile device. 
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Heuristic Evaluation Dashboard 


Nielsen’s 10 Usability Heuristics 

D 

2 

3 

D 

5 

6 

7 

8 

9 

10 

11 

12 

Visibility of system status 

5 

5 

5 

5 

4 

5 

4 

5 

4 

5 

3 

3 

Match between system and the real 
world 

4 

5 

5 

2 

3 

4 

4 

4 

5 

5 

3 

3 

User control and freedom 

5 

5 

5 

5 

5 

5 

4 

2 

5 

5 

4 

5 

Consistency and standards 

5 

5 

5 

5 

5 

5 

4 

5 

3 

5 

3 

5 

Error prevention 

5 

5 

3 

5 

4 

4 

1 

3 

2 

5 

4 

3 

Recognition rather than recall 

5 

5 

5 

3 

3 

3 

4 

5 

3 

5 

3 

5 

Flexibility and efficiency of use 

5 

5 

5 

4 

3 

5 

4 

5 

2 

5 

4 

5 

Aesthetic and minimalist design 

5 

5 

5 

2 

2 

5 

4 

5 

3 

5 

5 

4 

Hep users recognize, diagnose, and 
recover from errors 

5 

5 

5 

4 

1 

5 

3 

5 

5 

5 

5 

5 

Help and documentation 

5 

5 

5 

2 

3 

4 

4 

4 

3 

5 

4 

5 

Total 

49 

50 

48 

37 

34 

45 

36 

43 

35 

50 

38 

43 

User Experience Score 

98 

% 

100 

% 

96 

% 

74 

% 

66 

% 

90 

% 

72 

% 

86 

% 

70 

% 

100 

% 

E 

6 

'o 

86 

% 


UX MOU Requirements 

D 

2 

3 

D 

5 

6 

7 

8 

9 

10 

11 

12 

Privacy, security, free from marketing 

4 

4 

4 

5 

5 

4 

4 

3 

3 

4 

5 

4 

Understands offering available 

4 

5 

5 

4 

3 

4 

3 

4 

4 

4 

4 

4 

User control and access 

5 

5 

5 

5 

2 

5 

3 

2 

5 

5 

3 

5 

Meets expectations 

4 

5 

5 

4 

4 

4 

5 

4 

5 

5 

4 

4 

New technologies 

4 

5 

5 

4 

5 

4 

4 

4 

4 

5 

4 

4 

Transparency around ineligibility 

4 

4 

1 

4 

4 

4 

3 

3 

4 

5 

5 

5 

Total 

25 

28 

25 

26 

23 

25 

22 

20 

25 

28 

25 

26 

MOU Score 

83 

% 

i 

83 

% 

87 

% 

77 

% 

83 

% 

i 

67 

% 

i 

93 

% 

i 

87 

% 


Mobile Experience 

D 

2 

3 

D 

5 

6 

7 

Mobile optimized site 

5 

5 

4 

4 

4 

5 

3 

Mobile Experience Score 

100 

% 

100 

% 

80 

% 

80 

% 

80 

% 

100 

% 

60 

% 


100 100 40 

% % % 
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Scoring Methodology 

User Experience 

■ Review and rate task flow and user interface of each Free File member 
software site. Every problem identified is tagged with one often Nielsen’s 
heuristics and a 1-5 severity score. 

■ When multiple issues are found under one category, the lowest score 
determined is used. 

■ Individual scores are tallied and a percentage score determined. 

Mobile Optimized Site 

Evaluators used Developer Tools “toggle device toolbar” to inspect mobile 

performance. The following checklist was be used: 

■ If the website is optimized for mobile phone browsers (e.g., larger navigation 
buttons, reformatted content, optimized images) it will receive a “5.” Any 
website that does not provide an optimized site (for example, the user must 
zoom in to read content or complete a task) will receive a ”2” - this is 
considered a major usability issue, however the content is still accessible. 
Inaccessible content receives a severity score of “1”. 

■ If a vendor receives two or more heuristic review scores of “4” on mobile 
screens, the score is reduced to “4” 

■ If a vendor receives one or more heuristic review scores of “3” on mobile 
screens, the score is reduced to “3” 

UX MOU Requirements 

■ Review and rate task flow and user interface of each Free File member 
software site. Any issue identified that does not align with the users 
perspective on the MOU is identified, tagged and rated on a 1-5 severity score. 

■ When multiple issues are found under one category, the lowest score 
determined is used. 

■ Individual scores are tallied and a percentage score determined. 
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Nielsen’s 10 Usability Heuristics 

Jakob Nielsen's 10 general principles for interaction design 


Nielsen’s 10 Usability Heuristics 

Visibility of system 
status 

The system should always keep users informed about what is going on, 
through appropriate feedback within reasonable time 

Match between 
system and the real 
world 

The system should speak the users' language, with words, phrases and 
concepts familiar to the user, rather than system-oriented terms. Follow 
real-world conventions, making information appear in a natural and 
logical order 

User control and 
freedom 

Users often choose system functions by mistake and will need a clearly 
marked "emergency exit" to leave the unwanted state without having to 
go through an extended dialogue. Support undo and redo 

Consistency and 
standards 

Users should not have to wonder whether different words, situations, or 
actions mean the same thing. Follow platform conventions 

Error prevention 

Even better than good error messages is a careful design which prevents a 
problem from occurring in the first place. Either eliminate error-prone 
conditions or check for them and present users with a confirmation 
option before they commit to the action 

Recognition rather 
than recall 

Minimize the user's memory load by making objects, actions, and options 
visible. The user should not have to remember information from one part 
of the dialogue to another. Instructions for use of the system should be 
visible or easily retrievable whenever appropriate 

Flexibility and 
efficiency of use 

Accelerators -- unseen by the novice user - may often speed up the 
interaction for the expert user such that the system can cater to both 
inexperienced and experienced users. Allow users to tailor frequent 
actions 

Aesthetic and 
minimalist design 

Dialogues should not contain information which is irrelevant or rarely 
needed. Every extra unit of information in a dialogue competes with the 
relevant units of information and diminishes their relative visibility 

Hep users 
recognize, 
diagnose, and 
recover from errors 

Error messages should be expressed in plain language (no codes), 
precisely indicate the problem, and constructively suggest a solution 

Help and 
documentation 

Even though it is better if the system can be used without documentation, 
it may be necessary to provide help and documentation. Any such 
information should be easy to search, focused on the user's task, list 
concrete steps to be carried out, and not be too large 
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User Experience (UX) MOU Requirements 

How the MOU requirements are demonstrated via the user experience 


UX MOU Requirements 

Privacy, security, 
free from marketing 

User recognizes the system is secure and their privacy is protected. The 
system does not link to its paid offerings, nor does it request any 
unnecessary personal information, such as billing information (except 
when legitimately offering a state return fora fee). (Reference: 4.1.3vi; 
4.32.6; 4.19.3; 4.19.4) 

Understands 
offering available 

User feels confident the forms and schedules they use are supported by 
the system, and understands which forms or schedules are not 
accepted. They see whether or not they can file their state return for 
free. If free filing is not offered for their state, they understand there is a 
fee and see the cost. They see a prominent link back to the IRS.gov Free 
File Landing Page on the systems’ landing page. Priorto and during their 
experience, the user has a clear sense of their free customer service 
options. (Reference: 4.5.3; 4.5.4; 4.21.1; 4.32.1; 4.15.4; 4.32.7) 

User control and 

access 

A user has the option to complete their return at a later date. They may 
choose to print their return for free and is aware of the option to do so. 
User may see the status of their return and obtain a copy after electronic 
filing. (Reference: 4.15.4; 4.15.4; 4.15.3; 4.15.1) 

Meets expectations 

User links to expected free file site from IRS.gov. The system meets 
user's expectations for the name of the site. User understands that the 
system is a legitimate Free File site via an appropriate logo. (Reference: 
4.19.3; 4.33; 4.34) 

New technologies 

Users are offered new technologies that pre-populate forms, such as 
uploading a W-2, when available. (Reference: 4.36.2) 

Transparency 
around ineligibility 

A user understands thatthey are ineligible as early as possible in the 
process. They understand why they are ineligible when informed. The 
user clearly sees a link back to IRS Free File Landing Page so that they 
may explore other options. In addition to understanding they may return 
to IRS.gov, the user may see that they may complete their return for a 
fee. In addition, an ineligible user who attempts to use the system if 
offered the option to print out their return for a fee equivalent to the 
systems commercial offerings. (Reference: 4.15.2; 4.19.2 vi; 4.19.2 1; 

4.19.2 ii; 4.19.2 iii; 4.19.2 iv; 4.32.2; 4.32.5) 
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